🕷️️ Job Radar • SCRAPING

freelancer.com 2026-04-14 🟡

🔹 Build Structured Web Crawling System
👤 Client: 🇩🇪 Bruchsal, Germany Member since 2026-04-12
💰 Price: $25 / hr Average bid
🚩 Problem: Develop a clean, well-defined web crawling system to process 30,000 profiles per week, integrating with an existing Laravel admin dashboard.
📦 Existing: Laravel-based admin dashboard

Specifications:

[Target] Develop a crawler using Scrapy
[Method] Build a FastAPI interface for start/stop/status/statistics
[UI/UX] Not specified
[Stack] Python 3.11+, Scrapy, FastAPI, MariaDB, Redis, Docker, Wasabi (S3)
[Security] Implement hash-based change detection and soft-delete logic with a 3-month lifecycle
[Format] Provide short technical documentation

Workflow:

Set up development environment using Python 3.11+.
Develop Scrapy crawler for web scraping.
Implement FastAPI interface for managing the crawler (start/stop/status/statistics).
Integrate with MariaDB and Redis for data storage and caching.
Create media pipeline: download → local cache → upload to Wasabi (S3).
Implement hash-based change detection logic.
Add soft-delete functionality with a 3-month lifecycle.
Dockerize the system for deployment on an Unraid server.
Document the system and provide technical documentation.